Diagnostic Accuracy and Fine-Grained Image-Interpretation Performance of Multimodal Large Language Models on Bilingual Radiology Questions: Cross-Sectional Benchmark Study

Introduction

In recent years, large language models (LLMs) have provided unprecedented opportunities in radiology [1]. By leveraging robust natural language processing, LLMs have demonstrated immense potential in handling complex radiological text tasks, such as generating structured reports and extracting clinical information [2]. As LLMs evolve toward multimodal architectures, their applications are expanding from singular text interpretation to direct image analysis. However, this cross-modal transition faces significant challenges. While large models exhibit high-level reasoning when processing structured radiological text—capable of generating accurate differential diagnoses solely from textual histories [3,4]—their diagnostic accuracy often declines when presented directly with complex medical images. Two radiology studies evaluating GPT-4–family multimodal systems reported lower performance under image-based conditions than under text-based or text-rich conditions, indicating limitations in radiologic image interpretation for these systems under the settings evaluated [5,6]. This necessitates systematic testing to ascertain the true extent of their visual feature extraction and diagnostic proficiency.

Currently, global LLMs exhibit distinct developmental trajectories in multimodal visual recognition. Globally applied models, such as GPT-5.2 and Gemini 3 Pro, represent the forefront of this technology. The OpenAI GPT series has evolved from early external visual modules to native omni-modal architectures, achieving significant leaps in multimodal feature alignment [7]. The Gemini series, designed natively multimodal from its foundation, leverages long-context reasoning and cross-modal feature fusion to establish a leading position [8]. Concurrently, LLMs developed in China have made rapid breakthroughs. Doubao (ByteDance), a consumer-facing AI assistant powered by ByteDance’s Seed model family, has undergone deep visual cognitive enhancement tailored to the Chinese medical ecosystem, demonstrating high evolutionary efficiency [9]. Notably, Yuanbao (Tencent), a consumer-facing AI assistant evaluated with the DeepSeek-R1 option selected in this study, innovatively integrates the breakthrough DeepSeek-R1 architecture. By using large-scale reinforcement learning and chain-of-thought deep reasoning, it substantially enhances the model’s capacity for self-reflection in complex medical diagnosis [10,11]. Cross-platform comparisons between these emerging regional models and global general models help explore the impact of diverse training data ecosystems on model efficacy, providing valuable empirical references for evaluating their generalizability [12,13]. However, despite these architectural advancements, the diagnostic stability and visual recognition capabilities of these multimodal LLMs across diverse medical imaging modalities (eg, computed tomography [CT], magnetic resonance imaging [MRI], radiography, ultrasound, and nuclear medicine [NM]) remain largely unclear, highlighting a critical gap that necessitates systematic evaluation. The combined evaluation of multimodal LLMs developed by companies based in China and the United States within the same standardized bilingual benchmark, together with structured image-interpretation assessment across imaging modalities and anatomical systems, remains limited.

Medical multiple-choice questions (MCQs), encompassing both formal board-style examinations and clinical image challenges, serve as reliable tools for evaluating the clinical reasoning of large models [14-16]. Therefore, we used MCQs that covered bilingual contexts, multiple imaging modalities, and various anatomical systems with zero-shot prompting [14]. The purpose of this cross-sectional benchmark study was to compare the capabilities of different multimodal LLMs (Gemini 3 Pro, Doubao, GPT-5.2, Yuanbao, and Llama 4 Maverick) in medical image recognition and clinical diagnostic reasoning and to systematically analyze influencing factors. In addition to comprehensively analyzing the impact of these variables on overall accuracy, we uniquely deconstructed the visual reasoning process of image-based questions into 4 fine-grained dimensions: modality recognition, anatomical localization, lesion detection, and lesion localization. Through this approach, we aimed to accurately delineate the capability boundaries of current multimodal LLMs in radiology, providing solid empirical evidence for future targeted optimization.

Methods

Data Collection and Organization

This cross-sectional benchmark study collected a dataset comprising 298 MCQs in radiology, covering both Chinese and English contexts. The Chinese MCQs, which included both text- and image-based items, were sourced from the Chinese National Health Professional Technical Qualification Examinations (Associate Senior Radiology, Nuclear Medicine, and Ultrasound Medicine). To ensure comprehensive modality coverage, the English MCQs were obtained from publicly available educational sources, including the New England Journal of Medicine Image Challenge, BMJ Emergency Medicine Journal Image Challenge, Asian Nuclear Medicine Board sample questions, and Radiopaedia MCQs [15]. The question sources, including question IDs, and identifiable publication periods are summarized in Table S1 in Multimedia Appendix 1. Before model evaluation, all questions and images were reviewed to confirm the absence of direct patient identifiers, and no identifiable patient information was entered into the evaluated LLMs.

Two board-certified radiologists, each with more than 5 years of clinical experience, independently screened and reviewed all questions. The inclusion criteria were as follows: (1) a complete question stem, (2) at least four answer options with a single definitive correct answer, and (3) for image-based questions, a medical image deemed clear and diagnostic by 2 radiologists, derived from a single imaging modality. The exclusion criteria were as follows: (1) nonradiologic images (eg, gross pathology specimens), (2) questions containing video or dynamic images, and (3) mixed-modality questions containing 2 or more imaging modalities within the same item.

The questions finally included were categorized as text-based or image-based. All questions were labeled according to imaging modality (CT, radiography, MRI, ultrasound, and NM) and anatomical system (neurologic, respiratory, cardiovascular, digestive, genitourinary, and musculoskeletal; Figure 1).

‎

Ethical Considerations

This study used only publicly available examination questions and published educational materials without identifiable patient information. Under applicable national ethical review regulations in China [16], an ethics review was not required for this study; accordingly, no formal exemption determination, protocol number, and application number were issued.

Model Selection and Evaluation

Four proprietary multimodal LLMs were selected based on their visual processing capabilities and leading market positions as of November 2025: Gemini 3 Pro (Google), Doubao (ByteDance), GPT-5.2 (OpenAI), and Yuanbao (Tencent; with the DeepSeek-R1 option selected). Models lacking visual input support were excluded. The 4 proprietary multimodal LLMs were evaluated between December 14, 2025, and January 31, 2026, via the official web-based interfaces of each model, without API access [17]. Detailed model configurations and access conditions are provided in Table S2 in Multimedia Appendix 1. The user-facing reasoning settings were Auto mode for GPT-5.2, Pro mode for Gemini 3 Pro, Deep Thinking for Doubao, and DeepSeek-R1 with Deep Thinking for Yuanbao. To provide an open-weight reproducibility anchor alongside the 4 proprietary systems, a separate evaluation of Meta Llama 4 Maverick 17B 128E Instruct was conducted in September 2026 using the same 298 questions. The model was accessed through a dedicated fixed end point, provided by Together AI, with detailed deployment and inference configurations provided in Table S3 in Multimedia Appendix 1. To prevent contextual carryover and potential memory bias, each question was evaluated independently; a new conversation was initiated for each question in the 4 web-based systems, whereas Llama 4 Maverick was submitted as an independent API request. Default model parameters were maintained for the 4 web-based systems to simulate real-world clinical workflows, and a zero-shot prompting strategy was applied [18]. Zero-shot prompting was selected because our objective was to evaluate the baseline, out-of-the-box performance of the evaluated models without task-specific demonstrations or prompt optimization. Input configurations were standardized by question type. For text-based questions, the input comprised the complete stem, answer options, and task instructions. For image-based questions, the medical image was uploaded alongside the text. To minimize translation-induced bias and simulate authentic clinical scenarios, prompts were constructed in the original language of the question (Chinese or English; eg, “Please select the correct answer for the above question”; Figure 2).

‎

The primary evaluation metric was the binary accuracy (correct or incorrect) of the model’s final answer, defined as the proportion of questions for which the model-selected option matched the reference answer. For image-based questions, to further investigate visual reasoning performance, 2 radiologists jointly reviewed the generated reasoning content. They reached a consensus judgment for each outcome, and only the final consensus result was recorded for analysis. Scoring was performed across four binary dimensions: (1) modality (whether the model correctly identified the imaging technique, such as distinguishing CT from MRI), (2) anatomy (whether the model correctly identified the depicted anatomical region or organ), (3) lesion detection (whether the model explicitly identified the abnormality in the reasoning text), and (4) localization (whether the model correctly described the precise location of the lesion) [19]. To verify the reliability and reproducibility of the 4 image-interpretation outcomes, 2 radiologists independently and blindly evaluated the diagnostic outputs generated by 5 LLMs for 40 randomly selected image-based questions (approximately 20% of the 198 image-based questions). The 40 questions ×5 LLMs yielded 200 LLM responses. For each response, both physicians determined whether the output was correct or incorrect across 4 prespecified dimensions: modality recognition, anatomical recognition, lesion detection, and lesion localization. Agreement was assessed separately for each dimension, with 200 paired ratings per dimension. Both radiologists were blinded to each other’s ratings.

Statistical Analysis

For intermodel comparisons, the Cochran Q test was used to evaluate overall differences in accuracy among the 5 models. Accuracy estimates were reported with 95% Wilson CIs. Analyses were conducted on the overall dataset and stratified by language, question type (image-based vs text-based), imaging modality, and anatomical system. Given the influence of visual information on model reasoning, separate analyses based on modality and anatomical system were performed within the image- and text-based subgroups. If the global test indicated statistical significance, post hoc pairwise comparisons were conducted using the 2-sided exact McNemar test [20]. Pairwise effect sizes were reported as absolute differences in accuracy with 95% Newcombe CIs for paired proportions. For intramodel consistency, the chi-square test was used to assess the performance stability of each model across independent subgroups (eg, Chinese vs English questions). Yates continuity correction was applied to 2×2 comparisons of language and question type, whereas standard Pearson chi-square tests were used for comparisons across the 5 imaging modalities and 6 anatomical systems. The 4 image-interpretation outcomes were evaluated for image-based questions. Interrater reliability for the 4 image-interpretation outcomes was assessed using standard Cohen κ. To account for highly skewed marginal distributions in certain outcomes, prevalence-adjusted and bias-adjusted kappa (PABAK) was also calculated. The 95% CIs were computed for all point estimates. The strength of agreement was categorized according to the Landis and Koch criteria: <0.00, poor; 0.00‐0.20, slight; 0.21‐0.40, fair; 0.41‐0.60, moderate; 0.61‐0.80, substantial; and 0.81‐1.00, almost perfect. To control for type I error inflation from multiple pairwise comparisons, all post hoc P values were corrected using the Holm-Bonferroni method. All 298 questions received valid responses from all 5 models, with no missing, refused, excluded, or imputed observations. All statistical tests were 2-sided, and a P value of less than .05 was considered indicative of a statistically significant difference, with Holm-adjusted P values used for post hoc pairwise comparisons. Data analysis and figure generation were performed using Python software (version 3.13).

Results

Characteristics of MCQs

A total of 298 medical imaging questions were included. The dataset comprised 198 (66.4%) image-based questions and 100 (33.6%) text-based questions. In terms of language distribution, the dataset was balanced between Chinese (n=160, 53.7%) and English (n=138, 46.3%). Questions covered multiple imaging modalities with relatively even distribution: CT (n=71, 23.8%), radiography (n=65, 21.8%), MRI (n=60, 20.1%), ultrasound (n=53, 17.8%), and NM (n=49, 16.4%). Questions were distributed across 6 major anatomical systems to ensure comprehensive subspecialty coverage: digestive (n=61, 20.5%), neurologic (n=56, 18.8%), cardiovascular (n=49, 16.4%), respiratory (n=49, 16.4%), musculoskeletal (n=44, 14.8%), and genitourinary (n=39, 13.1%; Table 1).

Overall Accuracy and Stratified Analysis by Language and Question Type

Across the entire dataset, the overall accuracy of the 5 evaluated multimodal LLMs ranged from 72.5% (n=216) to 89.9% (n=268). Overall accuracy differed among the 5 LLMs (Cochran Q test, P<.001). Pairwise comparisons demonstrated that Gemini 3 Pro achieved higher overall accuracy (n=268, 89.9%, 95% CI 86.0%‐92.9%) than Doubao (n=235, 78.9%, 95% CI 73.9%‐83.1%; P<.001), GPT-5.2 (n=233, 78.2%, 95% CI 73.2%‐82.5%; P<.001), Yuanbao (n=216, 72.5%, 95% CI 67.2%‐77.2%; P<.001), and Llama 4 Maverick (n=216, 72.5%, 95% CI 67.2%‐77.2%; P<.001). No significant differences were detected among Doubao, GPT-5.2, Yuanbao, and Llama 4 Maverick (Table 2 and Figure 3A). Pairwise absolute accuracy differences with 95% Newcombe CIs are reported in Table S4 in Multimedia Appendix 1.

‎

For Chinese-language (n=160) questions, overall differences were observed among models (P<.001). Gemini 3 Pro (n=139, 86.9%, 95% CI 80.8%‐91.3%) outperformed GPT-5.2 (n=109, 68.1%, 95% CI 60.6%‐74.8%; P<.001), Yuanbao (n=107, 66.9%, 95% CI 59.3%‐73.7%; P<.001), and Llama 4 Maverick (n=107, 66.9%, 95% CI 59.3%‐73.7%; P<.001). The difference between Doubao (n=123, 76.9%, 95% CI 69.8%‐82.7%) and Gemini 3 Pro was not statistically significant. For English-language questions (n=138), overall differences were also observed (P<.001). Gemini 3 Pro (n=129, 93.5%, 95% CI 88.1%‐96.5%) achieved higher accuracy than Doubao (n=112, 81.2%, 95% CI 73.8%‐86.8%; P=.01), Yuanbao (n=109, 79.0%, 95% CI 71.4%‐85.0%; P<.001), and Llama 4 Maverick (n=109, 79.0%, 95% CI 71.4%‐85.0%; P<.001). GPT-5.2 (n=124, 89.9%, 95% CI 83.7%‐93.9%) also outperformed Yuanbao (P=.006) and Llama 4 Maverick (P=.01). No significant differences were detected between Gemini 3 Pro and GPT-5.2 (Table 2 and Figure 3A).

Regarding intramodel language comparisons, Yuanbao, Llama 4 Maverick, and GPT-5.2 achieved higher accuracy on English questions than on Chinese questions (P=.03, P=.03, and P<.001, respectively). No significant difference was detected for Doubao or Gemini 3 Pro (Figure 3B).

For image-based (n=198) questions, overall differences were observed among models (P<.001). Gemini 3 Pro (n=170, 85.9%, 95% CI 80.3%‐90.0%) outperformed Doubao (n=148, 74.7%, 95% CI 68.3%‐80.3%; P=.02), GPT-5.2 (n=142, 71.7%, 95% CI 65.1%‐77.5%; P<.001), Yuanbao (n=126, 63.6%, 95% CI 56.7%‐70.0%; P<.001), and Llama 4 Maverick (n=131, 66.2%, 95% CI 59.3%‐72.4%; P<.001). Doubao also outperformed Yuanbao (P=.02). For text-based (n=100) questions, overall differences were observed (P=.002). Pairwise comparisons showed that Gemini 3 Pro (n=98, 98.0%, 95% CI 93.0%‐99.4%) achieved higher accuracy than Doubao (n=87, 87.0%, 95% CI 79.0%‐92.2%; P=.03) and Llama 4 Maverick (n=85, 85.0%, 95% CI 76.7%‐90.7%; P=.002; Table 2 and Figure 3A).

Within-model comparisons showed that all 5 models achieved higher accuracy on text-based questions than on image-based questions (Yuanbao, P<.001; Doubao, P=.02; GPT-5.2, P<.001; Gemini 3 Pro, P=.002; Llama 4 Maverick, P<.001; Figure 3C).

Accuracy Analysis Based on Imaging Modality

In the overall dataset stratified by imaging modalities, model accuracy ranged from 66.2% to 91.7%. Overall differences among the 5 models were observed for CT (P<.001), MRI (P=.03), ultrasound (P=.01), and radiography (P<.001) questions. No significant overall difference was detected for NM (P=.08). Post hoc pairwise comparisons demonstrated that for CT, both Gemini 3 Pro (63/71, 88.7%, 95% CI 79.3%‐94.2%) and GPT-5.2 (58/71, 81.7%, 95% CI 71.2%‐89.0%) achieved higher accuracy than Yuanbao (47/71, 66.2%, 95% CI 54.6%‐76.1%; P=.001 and P=.009, respectively); Gemini 3 Pro also outperformed Llama 4 Maverick (49/71, 69.0%, 95% CI 57.5%‐78.6%; P=.02). For MRI, although the overall difference was significant (P=.03), no pairwise differences remained significant after Holm-Bonferroni correction. For ultrasound, Gemini 3 Pro (48/53, 90.6%, 95% CI 79.7%‐95.9%) achieved higher accuracy than GPT-5.2 (38/53, 71.7%, 95% CI 58.4%‐82.0%; P=.02) and Llama 4 Maverick (36/53, 67.9%, 95% CI 54.5%‐78.9%; P=.04). For radiography, Gemini 3 Pro (59/65, 90.8%, 95% CI 81.3%‐95.7%) and Doubao (58/65, 89.2%, 95% CI 79.4%‐94.7%) achieved higher accuracy than Llama 4 Maverick (45/65, 69.2%, 95% CI 57.2%‐79.1%; P=.01 and P=.02, respectively; Table 3 and Figure 4A).

‎

When restricted to image-based questions, overall differences were observed for CT (P<.001), radiography (P<.001), and ultrasound (P=.02) modalities. Pairwise comparisons showed that for CT questions, Gemini 3 Pro (44/51, 86.3%, 95% CI 74.3%‐93.2%) achieved higher accuracy than Yuanbao (29/51, 56.9%, 95% CI 43.3%‐69.5%; P<.001); GPT-5.2 (40/51, 78.4%, 95% CI 65.4%‐87.5%) also outperformed Yuanbao (P=.009). For ultrasound questions, although the overall difference was significant (P=.02), no pairwise differences remained significant after Holm-Bonferroni correction. For radiography, both Doubao (38/44, 86.4%, 95% CI 73.3%‐93.6%) and Gemini 3 Pro (38/44, 86.4%, 95% CI 73.3%‐93.6%) outperformed Llama 4 Maverick (26/44, 59.1%, 95% CI 44.4%‐72.3%; both P=.04). No significant overall differences were detected for MRI or NM (Table 3 and Figure 4B). For text-based questions, no evidence of an overall difference was detected among the 5 models across all 5 imaging modalities (CT, P=.28; radiography, P=.23; MRI, P=.13; ultrasound, P=.08; and NM, P>.99; Table 3 and Figure 4C).

Regarding intramodel performance, no significant difference in accuracy was detected among the 5 imaging modalities for any of the evaluated models (Yuanbao, P=.46; Doubao, P=.21; GPT-5.2, P=.26; Gemini 3 Pro, P=.96; and Llama 4 Maverick, P=.45; Figure 5).

‎

Accuracy Analysis Based on the Anatomical System

In the overall dataset stratified by the anatomical system, model accuracy ranged from 65.3% to 95.9%. Overall differences among the 5 models were observed for neurologic (P=.04), respiratory (P<.001), cardiovascular (P=.02), genitourinary (P=.04), and musculoskeletal (P=.02) systems. No significant overall difference was detected for the digestive system (P=.17). Post hoc pairwise comparisons demonstrated that for the respiratory system, Gemini 3 Pro (47/49, 95.9%, 95% CI 86.3%‐98.9%) achieved higher accuracy than Yuanbao (35/49, 71.4%, 95% CI 57.6%‐82.2%; P=.02) and Llama 4 Maverick (34/49, 69.4%, 95% CI 55.5%‐80.5%; P=.01). Although overall differences were detected for neurologic, cardiovascular, genitourinary, and musculoskeletal systems, no pairwise differences were identified after Holm-Bonferroni correction (Table 4 and Figure 6A).

‎

When restricted to image-based questions, overall differences were observed for neurologic (P=.04), respiratory (P=.002), and cardiovascular (P=.04) systems. Pairwise comparisons showed that for image-based respiratory questions, Gemini 3 Pro (28/30, 93.3%, 95% CI 78.7%‐98.2%) performed better than Yuanbao (17/30, 56.7%, 95% CI 39.2%‐72.6%; P=.03). No significant pairwise differences were detected in the neurologic or cardiovascular systems. For digestive, genitourinary, and musculoskeletal systems, no significant overall differences were detected (Table 4 and Figure 6B). For text-based questions, no overall differences among the 5 models were observed across all 6 anatomical systems (neurologic, P=.86; respiratory, P=.32; cardiovascular, P=.41; digestive, P=.32; genitourinary, P=.23; musculoskeletal, P=.09; Table 4 and Figure 6C).

For intramodel comparisons, no significant difference was detected in accuracy across the 6 anatomical systems for any evaluated model (Yuanbao, P=.82; Doubao, P=.52; GPT-5.2, P=.35; Gemini 3 Pro, P=.13; Llama 4 Maverick, P=.97; Figure 7).

‎

Given the relatively small number of questions in some modality- and anatomical-system subgroups, these subgroup estimates should be interpreted cautiously in conjunction with their 95% CIs.

Visual Reasoning Performance on Image-Based Questions

In the evaluation of visual recognition tasks related to imaging modality classification and anatomical structure identification, all 5 models demonstrated high accuracy ranging from 196 of 198 (99.0%, 95% CI 96.4%‐99.7%) to 198 of 198 (100.0%, 95% CI 98.1%‐100.0%). No significant between-model differences were detected for modality recognition or anatomical recognition (P=.56 and P=.25, respectively).

In contrast to the near-perfect performance in basic visual recognition, model accuracy decreased in more complex visual interpretation tasks, ranging from 68.2% to 88.4% for lesion detection and from 84.8% to 95.5% for lesion localization. For lesion detection, overall differences among models were observed (P<.001). Post hoc pairwise comparisons demonstrated that Gemini 3 Pro (175/198, 88.4%, 95% CI 83.2%‐92.1%) achieved higher accuracy than Yuanbao (135/198, 68.2%, 95% CI 61.4%‐74.3%; P<.001), GPT-5.2 (152/198, 76.8%, 95% CI 70.4%‐82.1%; P=.002), and Llama 4 Maverick (142/198, 71.7%, 95% CI 65.1%‐77.5%; P<.001). Additionally, Doubao (159/198, 80.3%, 95% CI 74.2%‐85.2%) outperformed Yuanbao (P=.003). No other pairwise differences remained significant after Holm-Bonferroni correction. For lesion localization, overall differences were also observed (P<.001). Pairwise comparisons demonstrated that Gemini 3 Pro (189/198, 95.5%, 95% CI 91.6%‐97.6%) outperformed Yuanbao (168/198, 84.8%, 95% CI 79.2%‐89.2%; P<.001), Doubao (171/198, 86.4%, 95% CI 80.9%‐90.5%; P=.004), and Llama 4 Maverick (172/198, 86.9%, 95% CI 81.5%‐90.9%; P=.004). No significant differences were detected among Yuanbao, Doubao, GPT-5.2, and Llama 4 Maverick in lesion localization (Table 5 and Figure 8).

‎

Interrater agreement was evaluated across 4 diagnostic dimensions: modality, anatomy, lesion, and localization. Observed agreement was very high, ranging from 97.5% to 99.5%. Because several dimensions exhibited highly skewed marginal distributions, PABAK was reported alongside the standard Cohen κ. PABAK ranged from 0.95 to 0.99, indicating almost perfect agreement across all dimensions (Table S5 in Multimedia Appendix 1).

Discussion

Principal Findings

This cross-sectional benchmark study evaluated the performance of 5 multimodal LLMs in answering medical imaging MCQs covering bilingual contexts, dual question types (image and text), multiple imaging modalities, and various anatomical systems. Our primary findings were as follows: first, Gemini 3 Pro achieved the highest overall accuracy; second, all evaluated models performed better on text-only questions than on image-based questions; finally, an in-depth analysis of visual dimensions revealed that although multimodal LLMs achieved near-perfect accuracy in basic visual recognition (eg, recognizing modality and anatomy), their performance declined and varied greatly among models in lesion detection and localization tasks involving more fine-grained lesion-level interpretation.

Comparison With Prior Work

Regarding overall performance and language stratification, Gemini 3 Pro demonstrated a clear advantage, while no significant overall differences were detected among Doubao, GPT-5.2, Yuanbao, and Llama 4 Maverick. The optimal comprehensive performance of Gemini 3 Pro may be related to its underlying native multimodal architecture [8]. Unlike traditional spliced visual models, Gemini natively integrates text and image data from the ground up. This architecture may facilitate information retention and reduce cross-modal loss when extracting complex medical imaging features and aligning them with clinical histories. Furthermore, its advantage in long-context reasoning may support its global logical deduction capabilities when handling complex medical MCQs with long stems [21]. In the subgroup analysis of language preference, we observed that GPT-5.2, Llama 4 Maverick, and Yuanbao achieved higher accuracy on English questions than on Chinese questions. This performance discrepancy is likely related to the composition of the corpora they were exposed to during pretraining. As noted by Bhayana [1], the pretraining data of current mainstream LLMs primarily rely on massive amounts of unstructured text from the internet, where high-quality medical literature, open-source imaging databases, and medical guidelines are overwhelmingly dominated by the English language. Notably, as a representative regional model, Doubao demonstrated an overall accuracy comparable to that of GPT-5.2 and achieved performance with no evidence of a difference from Gemini 3 Pro on Chinese-language questions. This pattern may be consistent with potential benefits of alignment or fine-tuning with high-quality localized medical corpora for language-specific clinical tasks [9]. The inclusion of Llama 4 Maverick also provides an open-weight reproducibility reference alongside the proprietary web-based systems. Previous medical studies have demonstrated the competitive performance of Llama-family models: Adams et al [22] reported that Llama 3 70B performed comparably to leading proprietary models on radiology board-style questions, while Fast et al [23] found Llama 3 70B to be among the highest-performing models in a medical benchmark spanning 13 specialties. These findings support our selection of Llama 4 Maverick as a representative open-weight comparator; its fixed model revision and documented inference configuration further enable evaluation under reproducible model conditions.

An important finding of this study is that the accuracy of all evaluated models on image-based questions was lower than that on text-only questions. This result is highly consistent with several recent studies evaluating the radiological capabilities of large models. For example, Atakır et al [5] found in their evaluation of ChatGPT-4o that its accuracy was highest under conditions with rich text input, while its performance was limited in pure image tasks. Hayden et al [6] also observed a similar “text-outperforms-image” phenomenon when evaluating the American College of Radiology question bank. A possible reason for this gap is that the logical reasoning capabilities of current LLMs in natural language processing are highly mature; however, within a multimodal architecture, the mapping process of aligning complex medical image pixel features to the text space may remain challenging. Recent studies indicate that limited sensitivity to subtle grayscale contrast may lead visual encoders to lose critical spatial and contextual information when extracting fine-grained features, such as lesion margins [3,24]. Prior studies suggest that models may derive substantial diagnostic information from textual clinical details, whereas performance can decline when visual information must be interpreted more directly [5]. The performance of Yuanbao in this evaluation illustrates the observed gap between strong text-oriented reasoning and more variable image-related performance in some multimodal systems. Although Yuanbao was evaluated with the DeepSeek-R1 option selected, renowned for its exceptional logical reasoning capabilities, its performance on image-based questions (particularly lesion detection) was substantially constrained. This may reflect differences in visual processing or cross-modal alignment, although the underlying mechanism cannot be determined from the present benchmark. More generally, limitations in visual feature extraction could constrain downstream reasoning when multimodal systems process complex radiological images [25].

In the evaluations across different imaging modalities and anatomical systems, our intramodel analysis showed that all models maintained stable accuracy when dealing with different modalities (CT, radiography, MRI; Yuanbao, P=.46; Doubao, P=.21; GPT-5.2, P=.26; Gemini 3 Pro, P=.96; and Llama 4 Maverick, P=.45) and systems (Yuanbao, P=.82; Doubao, P=.52; GPT-5.2, P=.35; Gemini 3 Pro, P=.13; and Llama 4 Maverick, P=.97). Given the relatively small sizes of some modality and anatomical-system subgroups, these findings should be interpreted cautiously and do not establish equivalent performance across subgroups. However, in intermodel comparisons, significant pairwise differences involving Gemini 3 Pro were observed in CT, radiography, ultrasound, and the respiratory system. This performance discrepancy may reflect the different emphases of these models in their multimodal pretraining strategies and specialty data coverage. First, at the modality level, CT requires interpretation of complex cross-sectional spatial anatomy, radiography involves superimposed projectional structures, while ultrasound demands the recognition of nonstandardized planes and high-noise acoustic textures. Currently, most multimodal LLMs face bottlenecks in aligning such modality-specific physical features [24]. The observed differences across these modalities may reflect differences in visual training data or multimodal alignment. Second, at the anatomical system level, the respiratory system was the only subgroup in which significant pairwise differences remained after multiple-comparison corrections. This system-specific difference is consistent with the view of Mukherjee et al [26]: when faced with medical imaging characterized by high professional barriers, the reasoning ceiling of a model is deeply constrained by the distribution of its underlying specialty training data. This difference may also stem from variations in model-specific training and alignment strategies [27].

The stratified analysis of higher-order visual dimensions for image-based questions further characterizes the specific limitations of multimodal LLMs in radiological visual interpretation. In basic macroscopic visual tasks, such as modality recognition and anatomical recognition, all models reached a ceiling effect (approaching 100%). However, when progressing to higher-order tasks such as lesion detection and localization, model performance showed noticeable divergence, and overall accuracy dropped. This suggests a limitation in fine-grained image interpretation, although the underlying architectural mechanisms were not directly evaluated in this study. They can easily identify that an image is a head CT while showing weaker performance in tasks requiring more fine-grained local lesion interpretation. In real-world radiological diagnosis, identifying lesion margins, subtle density changes, and precise spatial adjacency relationships is crucial. Limitations in fine-grained visual perception may lead large models to produce visual hallucinations or miss diagnoses in complex medical images [1]. Unlike previous studies that treated visual interpretation largely as a “black box” by focusing solely on overall accuracy, our fine-grained deconstruction uniquely illuminates the specific sources of these errors. Consequently, this finding suggests that future model development may benefit from greater attention to local feature perception.

Limitations

This study had several limitations. First, although we conducted a systematic evaluation of the models using a standardized benchmark, the sample size was relatively limited (N=298), and some modality- and anatomical-system subgroups contained relatively few questions, which may limit the precision and generalizability of subgroup estimates. Second, because the test questions originated from past examination banks and publicly available educational sources, prior exposure or benchmark contamination cannot be excluded, and differential exposure across models may have influenced cross-model comparisons. Targeted spot-checks confirmed that some public-source items were searchable online verbatim or near-verbatim. Third, because image-based questions also contained textual information and no formal input-ablation analysis was performed, the independent contribution of visual input could not be isolated. Fourth, this experiment used standardized zero-shot prompting and did not evaluate performance under optimized prompting strategies. Fifth, LLMs undergo extremely rapid iterative updates. The 4 proprietary web-based systems remain subject to vendor updates, limiting independent reproducibility. Although the fixed Llama 4 Maverick revision provides an open-weight reproducibility anchor, it was evaluated later than the 4 proprietary systems, and the findings should still be regarded as a time-stamped benchmark.

Conclusions

Multimodal LLMs, represented by Gemini 3 Pro, demonstrated outstanding overall performance and broad cross-modal adaptability in medical imaging MCQs, highlighting the potential of multimodal LLMs as tools for clinical education and decision support. However, performance varied across languages, question types, and specific image-interpretation tasks, with lower and more variable performance in lesion detection and localization than in modality and anatomical recognition. Future research and model development should focus on further improving fine-grained lesion-level image interpretation and validating these systems on independent clinical imaging datasets before clinical implementation.